Methods and systems for efficiently moving data between nodes in a cluster

ABSTRACT

Exemplary embodiments provide methods, mediums, and systems for efficiently moving data between cluster nodes. Upon receiving a request to read or write data at a first cluster node that is in communication with a client, the first node effects the transfer to or from a second cluster node. The transfer is carried out using a combination of remote data memory access (“RDMA”), or a similar technique that bypasses a part of the network stack, and transport control protocol (“TCP”), or a similar technique that does not bypass a part of the network stack. The data is transferred using RDMA, while certain control messages are sent using TCP. By combining RDMA content transfers and TCP control messages, data transfers can be carried out faster, more efficiently, and with less processing overhead. Other embodiments are described and claimed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/840,512, entitled “METHODS AND SYSTEMS FOR EFFICIENTLY MOVING DATABETWEEN NODES IN A CLUSTER,” filed Aug. 31, 2015, which claims priorityto, and benefit of, U.S. Provisional Patent Application Ser. No.62/199,633, entitled “METHODS AND SYSTEMS FOR EFFICIENTLY MOVING DATABETWEEN NODES IN A CLUSTER,” filed on Jul. 31, 2015. The subject matterof U.S. patent application Ser. No. 14/840,512 and U.S. ProvisionalPatent Application Ser. No. 62/199,633 are hereby incorporated herein byreference in their respective entireties.

BACKGROUND

Computing devices may be organized into clusters that provide servicesto an accessing computing device, known as a client. Devices that makeup the cluster are referred to as nodes of the cluster. The presentapplication is addressed to the problem of efficiently moving databetween the nodes of a cluster.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting an example of a system including acluster and a client device suitable for use with exemplary embodimentsdescribed herein.

FIG. 2 is a block diagram depicting an exemplary electronic computingdevice.

FIG. 3A is a block diagram depicting a comparative example of a localread performed in the system of FIG. 1.

FIG. 3B is a block diagram depicting a comparative example of a remoteread performed using TCP in the system of FIG. 1.

FIG. 3C is a block diagram depicting a comparative example of a remoteread performed using RDMA in the system of FIG. 1.

FIG. 4 is a block diagram depicting an example of a remote readperformed according to an exemplary embodiment.

FIG. 5 is a block diagram depicting an example of a remote writeperformed according to an exemplary embodiment.

FIGS. 6A-6D depict exemplary data structures suitable for use withexemplary embodiments.

FIG. 7 is a data flow diagram depicting exemplary communication paths incluster.

FIG. 8 is a flowchart depicting an exemplary method performed by aclient device to request that a cluster perform an I/O operation.

FIGS. 9A-9C are flowcharts depicting an exemplary method performed by acluster point of contact node to interact with a client requesting anI/O operation and a remote node involved in the I/O operation.

FIGS. 10A-10B are flowcharts depicting exemplary methods performed by acluster remote node to interact with a sending node requesting an I/Ooperation.

DETAILED DESCRIPTION

Computing devices may be organized into clusters that provide servicesto an accessing computing device, known as a client. Devices that makeup the cluster are referred to as nodes of the cluster. Although thecluster may be made up of multiple distinct nodes, it may be presentedto the client as a single entity. The client may be assigned to interactwith a particular node of the cluster, or may interact with whichevernode is most convenient. The node with which the client interacts isreferred to herein as the point of contact (PoC) node for cluster. Theclient contacts the PoC node with requests for different services, andthe PoC node contacts other nodes that may be needed for the requestedservice and abstracts away the internal cluster interactions so that theclient is presented with a unified experience.

One such service that the cluster may provide is data storage. A clientmay access data in the cluster through input/output (“I/O”) requests,such as requests to read data from a storage system in the cluster, orwrite data to a storage system in the cluster.

Although the client interacts with the PoC node in the cluster, theclient may send a request to read data that is stored on a storagevolume that is located remotely from the PoC node. In another example,the client may request that data be stored in the cluster (a “write”request), and the PoC or a cluster administrator may determine that thedata should be stored on a storage volume remote from the PoC node.

In these cases, the PoC node may manage interactions between the clientand the remote storage volume. For example, the PoC node may send datato a remote node associated with the storage volume so that the remotenode can write the data to the storage volume. Alternatively or inaddition, the PoC node may forward a read request to the remote nodewhen a client requests data stored on a volume associated with theremote node, and may relay the retrieved data back to the client.

Exemplary embodiments provide methods, mediums, and systems forefficiently moving data between cluster nodes. Upon receiving a requestto read or write data at a first cluster node that is in communicationwith a client, the first node effects the transfer to or from a secondcluster node. The transfer is carried out using a combination of remotedata memory access (“RDMA”), or a similar technique that bypasses a partof the network stack, and transport control protocol (“TCP”), or asimilar technique that does not bypass a part of the network stack. Thedata is transferred using RDMA, while certain control messages andmetadata are sent using established TCP communication paths. The RDMAportion of the technique is performed in a counterintuitive manner: inorder to write data, an RDMA read command is issued, and in order toread data, an RDMA write command is issued.

Such an approach has advantages over alternative options. As compared tousing a TCP-only method of data transfer, exemplary embodiments conserveprocessing resources by avoiding the need to perform network overheadoperations associated with the bypassed portions of the network stack.For example, in conventional TCP-based data transfers, a chain ofbuffers is allocated as the data moves through the network stack. Atvarious protocol layers, the stack may be locked, and latency may beincurred due to protocol-based flow control.

As compared to using an RDMA-only approach in which both the data andcontrol messages are sent using RDMA, exemplary embodiments avoid theneed to provide an out-of-bound mechanism to cause the receiving node toact on the data. Because RDMA is a one-sided communication, RDMA doesnot notify the recipient that the data transfer has been completed.Thus, in an RDMA-only approach, RDMA control messages must beimplemented in conjunction with an out-of-band mechanism that causes therecipient node to act appropriately in response to the loaded data(which is inefficient), or some type of polling (which consumesprocessing resources and network resources). Such an out-of-bandmechanism or polling-based solution can be avoided by employing theexemplary embodiments described.

Moreover, it cannot necessarily be guaranteed that all nodes in thecluster will implement RDMA. Even if RDMA is implemented on a node, theRDMA communication pathways may become constrained, thus rendering itdifficult or impossible to transfer data using RDMA. By providing asystem that uses a combination of RDMA and TCP, the data transfer can beeffectuated by TCP if RDMA is unavailable.

For illustration purposes, an exemplary environment and apparatus inwhich exemplary embodiments may be employed are next described withreference to FIGS. 1 and 2.

Overall System and Environment

FIG. 1 depicts an example of a cluster 10 containing two nodes: a pointof contact node 12 and a remote node 14. A node refers to an electronicdevice that provides services, such as data storage or processing, onbehalf of the cluster. Although two nodes are depicted for ease ofdiscussion, a cluster 10 may be made up of any number of nodes. As shownin FIG. 1, each node 12, 14 is associated with a storage volume 16, 18(respectively). The storage volumes 16, 18 store data on behalf of theirassociated nodes 12, 14.

The cluster 10 is presented to outside users, such as the client 20, asa unitary entity. That is, the client 20 interacts with the cluster 10as though the cluster 10 were a single device, even though the clusteris made up of multiple devices. This allows the client 20 to benefitfrom the services provided by the nodes of the cluster 10, withoutneeding to know the internal structure or workings of the cluster 10.

In order to send tasks to the cluster 10 and receive results from thecluster 10, the client 20 communicates with one of the nodes of thecluster (the point of contact node 12) using one or more communicationlinks 22. The communication links 22 may be, for example, TCPcommunication links.

Although the client 20 communicates directly with the point of contactnode 12, it is not necessarily the case that the client 20 will requestthat the cluster 10 perform a task that will be handled by the point ofcontact node 12. For example, a load balancer in the cluster 10 maydetermine that the task should be handled by the remote node 14.Alternatively or in addition, the client 20 may request data from thecluster 10 that is stored on the storage volume 18 associated with theremote node 14. In order to perform the task or retrieve the data, thepoint of contact node 12 may communicate with the remote node 14 usingone or more intra-cluster communication links 24. The intra-clustercommunication links 24 may be, for example, TCP communication links orRDMA communication links.

FIG. 2 depicts an example of an electronic device suitable for use as anode in the cluster 10. For purposes of discussion, a point of contactnode 12 is depicted in FIG. 2, although a similar device having similarcomponents may be used as a remote node 14 or a client 20.

As noted above with respect to FIG. 1, the point of contact node 12 maybe associated with a storage volume 16. The storage volume 16 may beintegrated with the point of contact node 12 or may be communicativelycoupled to point of contact node 12. The point of contact node 12 may beconfigured to perform read and write operations on the storage volume 16by sending appropriate read or write commands to the storage volume 16.Examples of storage volumes 16 include partitions on hard disk drives(HDD and solid state drives (SSD). The storage volume 16 may include oneor several physical drives, and may be arranged into a redundant arrayof independent disks (RAID) arrangement.

The point of contact node 12 may include a network adapter 26 forcommunicating using the communication links 22, 24. The network adapter26 may be, for example, a wired adapter such as a network interfacecontroller for establishing a wired connection to a computer network, afiber optic interface for connecting to a fiber optic network, a cableinterface for connecting to a cable television network, a telephone jackfor connecting to a telephone network, or a power-line interface forconnecting to a power-line communications network. Alternatively or inaddition, the network adapter 26 may be a wireless adapter such as aradio transmitter/receiver that modulates and demodulateselectromagnetic signals. Examples of wireless network adapters 26include devices communicating through short-wavelength ultra-highfrequency (UHF) radio waves.

The network adapter 26 may optionally be configured with RDMA logic 28for communicating using the RDMA protocol. By configuring the networkadapter 26 with RDMA logic, the network adapter 26 may be capable ofreading data from, and writing data to, other devices in the cluster 10without using the processor 32 or the operating system 40.

The point of contact node 12 may use the network adapter 26 tocommunicate with other devices via a network 30. The network 30 may be apublic network such as the Internet, or a private network such as aprivate Data ONTAP® network provided by NetApp, Inc. of Sunnyvale,Calif.

The point of contact node 12 may include a processor 32 for performingmathematical and/or logical operations on behalf of the point of contactnode 12. The processor 32 may be a Central Processing Unit (CPU) havingone or more processing cores, one or more coprocessors, and/or on-chipcache. Examples of processors 32 include the Celeron®, Pentium®, Core™,and Atom™ families of processors from Intel Corporation of Santa Clara,Calif., the Accelerated Processing Unit (APU) and Central ProcessingUnit (CPU) processors from Advanced Micro Devices (AMD), Inc. ofSunnyvale, Calif., the Snapdragon™ family of processors from QualcommTechnologies, Inc. of San Diego Calif., and the Cortex® family ofprocessors from ARM Holdings, PLC of Cambridge, England.

The point of contact node 12 may include a memory 34 for holding data,instructions, and other information for use by the other components ofthe point of contact node 12. The memory 34 may be a non-transitorycomputer readable medium. For example, the memory 34 may be solid-statestorage media such as flash memory and/or random access memory (RAM).

The memory 34 may include a storage buffer 36 for temporarily storingdata until the data can be processed by the processor 32 or transmittedusing the network adapter 26. The storage buffer 36 may be, for example,a designated region in the memory 34 and may be embodied as a circularbuffer. The processor 32 or the RDMA logic 28 may allocate space in thememory 34 to serve as a storage buffer 36, and may deallocate the spaceoccupied by the storage buffer 36 when the storage buffer 36 is nolonger needed in order to conserve memory resources.

The memory 34 may be configured with Input/Output (“I/O”) processinglogic 38. The I/O processing logic may include instructions that, whenexecuted by the processor 32, cause the processor 32 to perform any ofthe methods described herein. For example, the processing logic 38 mayimplement any of the functionality or capabilities depicted in FIGS.4-10B.

The memory 34 may also store an operating system 40 for controlling thepoint of contact node 12. Examples of operating systems 40 include theWindows® family of operating systems by Microsoft, Inc. of Redmond,Wash., or the Data ONTAP® operating system by NetApp, Inc. of Sunnyvale,Calif.

The environment and apparatuses of FIGS. 1 and 2 may be used toimplement the exemplary embodiments described herein. As noted above,these embodiments make use of a combination of RDMA and TCP (or similarprotocols) and carry advantages over possible alternative approachesrelying on RDMA or TCP alone. To better illustrate the advantages of theexemplary embodiments, comparative examples employing TCP and RDMA aloneare next described with reference to FIGS. 3A-3C. In the examples thatfollow, numbers in parentheses refer to steps as illustrated in therespective Figures, although it is noted that steps may be performed ina different order in some circumstances.

Comparative Examples

FIG. 3A is a block diagram depicting an example of a local read in thecluster. A local read occurs when the client 20 requests access to data42 that is stored on the storage volume 16 that is locally accessible tothe point of contact node 12. This is usually the most efficient way toaccess the data 42, because the data 42 does not need to be transferredamong the nodes of the cluster.

First (1), the client 20 formulates a read request 44 and sends the readrequest to the point of contact node 12 over a communication link 22that implements TCP. The point of contact node 12 receives the requestand performs (2) a local read operation 46 to retrieve the data 42 fromthe local storage volume 16. After retrieving the data 42 from the localstorage volume 16, the point of contact node 12 transmits (3) therequested data in the form of TCP data packets 48 at step.

In order to effect the transfer of the data according to TCP, someamount of network overhead 50 is incurred (4). When the data 42 isreceived at the point of contact node 12, the data is processedaccording to a series of steps referred to as a network protocol stack.These steps may involve a series of actions performed by the networkadapter 26, by software applications run by the point of contact node12, and by the operating system 40, among other possibilities. Thenetwork stack is typically divided into multiple layers; for example,the popular Open Systems Interconnection (“OSI”) model of computernetworking involves seven layers, from the low-level actions performedby the physical transmission links to the high level operationsperformed by the operating system and individual software applicationsrunning on the operating system. The network overhead 50 may involve theallocation of one or more storage buffers 36 at different layers of theprotocol stack, and the use of resources of the processor 32 to preparethe data at each layer and ensure that the data has been successfullyreceived, among other possibilities.

The amount of network overhead 50 involved is partially dependent on thesize of the maximum transmission unit (MTU) associated with thecommunication links 22, which defines the maximum allowable size of dataunits that are placed onto the communication links 22. Typically, datafor transmission on a network is broken down into smaller units calledpackets. The size of each packet is dictated by the MTU of thecommunication links 22 over which the data is sent. For example, if thedata 42 is 4000 bytes in size, but the MTU of the communication links 22is 1500 bytes, then the data 42 will need to be broken down into atleast three packets. If the communication links 12 have a relativelysmall MTU (meaning that only a relatively small amount of data can beencoded into each packet), then the point of contact node 12 may need tobreak the data 42 into relatively more packets than if the MTU werelarger. The transmission of each packet incurs a certain amount ofnetwork overhead 50, which means that the network overhead 50 will scaleup as the MTU is made smaller. Some of the network overhead 50 isinherent to any transmission of the data, while some is due to actionsperformed as part of the TCP protocol.

In the local read technique depicted in FIG. 3A, the network overhead 50associated with the use of TCP is only incurred once, in thetransmission of the data from the point of contact node 12 to the client20. However, when the data 42 is not located on the local storage volume16 associated with the point of contact node, the network overhead 50 iscompounded.

For example, consider the example depicted in FIG. 3B, in which the data42 is stored on the storage volume 18 associated with the remote node14. This is referred to as a remote read.

In the example depicted in FIG. 3B, the remote read is implementedsolely with TCP. First (1), a read request is transmitted from theclient 20 to the point of contact node 12 over the communication link22. Upon determining that the requested data is not located at the localstorage volume 16, but rather is located at a remote storage volume 18,the point of contact node 12 generates an internal read request 44 and(2) transmits the read request to the remote node 14. The remote node 14performs (3) a local read operation 46 to retrieve the data 42 from thestorage volume 18 associated with the remote node 14. The remote node 14then transmits (4) the requested data back to the point of contact node12 using as a series of TCP packets 48. This step incurs (5) networkoverhead 50 associated with the use of TCP.

Upon receiving the data 42 from the remote node, the point of contactnode 12 transmits (6) the data back to the client over the communicationlink 22 as a series of TCP packets 48. This step also incurs (7) networkoverhead 50 associated with the use of TCP.

As can be seen from this example, performing a remote read doubles theamount of network overhead 50 incurred as a result of the use of TCP, ascompared to the local read of FIG. 3A. FIG. 3B depicts a relativelysimple example in which the point of contact node 12 is able to directlycontact the remote node 14, but this will not necessarily occur in allsituations. In some circumstances, it may be necessary to communicatewith one or more intermediate nodes in order to reach the remote node14. At each “hop” in the network, the amount of network overhead 50 isincreased.

As an alternative, data may be transferred between nodes in the clusterusing RDMA, which allows the network overhead 50 associated with the useof TCP to be avoided. This is because RDMA reads and writes datadirectly from/to remote memory, without the need to involve theoperating system 40 of the node associated with the remote storagevolume. FIG. 3C depicts an example of a remote read performed using RDMAalone (within the cluster, as the client 20 typically communicates withthe cluster using TCP).

The client 20 formulates a read request 44 and transmits (1) the readrequest 44 to the point of contact node 12 using TCP. The point ofcontact node 12 determines that the requested data is not located on thelocal storage volume 16, but rather is located at a remote storagevolume 18. The point of contact node then formulates a read request andtransmits (2) the request to the remote node 14 using RDMA. Because thepoint of contact node 12 may not know the memory location on the remotestorage volume 18 at which the data is stored, this step (2) may involveadditional overhead. For example, the point of contact node 12 may issueout of bound RDMA messages that inquire where the data is located, andthe remote node 14 may respond with out of bound RDMA messages providingthe address of the data. At that point, the point of contact node 12 mayissue an RDMA read request 50 to read the data at the identifiedaddress.

Upon receiving an RDMA read request 50, the network adapter 26 of theremote node 14 recognizes the RDMA read request and bypasses some of thelayers of the network stack. The remote node 14 retrieves (3) the data42 from the storage volume 18, and places (4) the data 42 directly intothe memory 34 of the point of contact node 12 using an RDMA datatransfer 52. This RDMA transfer also avoids some layers of the networkstack at the point of contact node 12.

Problematically, RDMA does not provide any mechanism to notify therecipient of data (the point of contact node 12, in this case) that thetransfer has been completed. Accordingly, the point of contact node 12and/or the remote node 14 must implement (5) some sort of out of boundmechanism. An out of band mechanism is a mechanism that operates on topof, or in addition to, the RDMA messages that actually effect thetransfer. For example, custom RDMA send/receive messages could be usedto inform the point of contact node 12 that the RDMA transfer has beencompleted. This typically requires additional messages beyond thoserequired to effect the data transfer, and may also require custom logicon the point of contact node 12 that recognizes the out of boundmessages and causes the point of contact node 12 to take appropriateactions in response. Because each RDMA operation requires an out of bandfollow-up message, the advantages of using RDMA are diminished.

Alternatively, after sending the RDMA read request 50 in the secondstep, the point of contact node 12 could begin polling the remote node14 to ask whether the transfer had been completed. This solution isrelatively inefficient, however, due to the processing and networkresources that are used in the polling.

Upon learning that the data 42 has been loaded into the memory 34 of thepoint of contact node 12, the point of contact node 12 transmits (6) thedata back to the client over the communication link 22 as a series ofTCP packets 48. This step incurs (7) network overhead 50 associated withthe use of TCP.

The benefit to this arrangement is that network overhead 50 associatedwith the use of TCP is not incurred in intra-cluster communications. Ithas the disadvantage, however, of requiring the above-noted out of boundor polling mechanism.

Next, consider a similar example (not illustrated for brevity) where theclient 20 requests that data be written to the volume 18 associated withthe remote node 14. In this case, the point of contact node 12 couldinform the remote node 14 that a write request was imminent using an outof bound RDMA message. The remote node 14 could allocate a buffer at theremote node 14 and send out of bound RDMA message informing the point ofcontact node 12 of the address where the buffer is located. The point ofcontact node 12 could then perform and RDMA write command and write thedata to the remote node's buffer. Once the transfer is complete, thepoint of contact node 12 would need to inform the remote node 14, againusing an out of bound RDMA message (or by employing a pollingmechanism), that the information has been transferred and that theremote node 14 should take appropriate steps to write the transferreddata to the volume 18 associated with the remote node 14.

As can be seen in these examples, each transfer involves the use of anumber of out of bound messages, polling, or both. As compared to theseexamples, embodiments of the present invention use RDMA to effect thetransfer of data content in a unique way, along with TCP to effect thetransfer of metadata and control messages. This has the advantage ofavoiding much of the intra-cluster network overhead associated with theuse of TCP, while also avoiding the need to deploy an out of bound orpolling mechanism to determine when a transfer has completed.

Exemplary Embodiments

FIG. 4 provides an example of a read operation performed according to anexemplary embodiment. First, the client 20 issues (1) a read request 44to the point of contact node 12 using TCP. The read request 44 requeststhat data 42 that is stored on a volume 18 associated with a remote node14 be returned to the client 20.

In response to receiving the read request 44, the point of contact nodepreallocates (2) a buffer 36 to hold the data 42. The point of contactnode 12 then sends (3), via TCP, a request 50 to the remote node. Therequest 50 includes the address of the buffer. Because the bufferaddress is relatively small, the request 50 can usually be transmittedas a single TCP packet, which limits the amount of TCP overheadassociated with this step.

Upon receiving the request, the remote node 14 performs (4) a local readoperation 46 to retrieve the data 42 from the volume 18. The data 42typically includes content and metadata. The remote node 14 separatesthe content from the metadata.

In the comparative example depicted in FIG. 3C, all of the data 42,including the content and the metadata, would be transferred using RDMA,and additional control messages would be sent to confirm that the data42 had been transferred. In contrast, in the exemplary embodimentdepicted in FIG. 4, the remote node 14 retrieves the buffer address fromthe previously-received TCP request 50, and uses an RDMA write operationto write (5) the content of the data 42 directly to the buffer 36 of thepoint of contact node 12. The metadata from the data 42 is then loadedinto a response message 52, which is sent (6) via TCP to the point ofcontact node 12. Because metadata is typically relatively small comparedto the content of the data 42, the metadata can often be transmitted asa single TCP packet (or at least relatively few TCP packets, if themetadata were to exceed the MTU of the communication link 24). Thissingle TCP packet serves as an acknowledgement that the data has beentransferred via RDMA, and therefore provides both confirmation and themetadata that is necessary for reconstructing the data 42, without theneed to resort to an out of bound mechanism or polling mechanism.

Upon receiving the read response 52, the point of contact node 12reconstructs the data 42 by combining the content received via RDMA withthe metadata received via TCP. The data 42 is then returned (7) to theclient 20 via TCP. Once the data is transferred, the point of contactnode 12 can safely deallocate (8) the buffer 36.

Using this technique, the intra-cluster network overhead is extremelylimited (only two TCP packets, at steps 3 and 6 in this example), thusaddressing the problems inherent in the TCP-only approach. Moreover, noout of band mechanism or polling mechanism needs to be employed, thusaddressing the problems with the RDMA-only approach.

A similar idea can be employed in the case of a remote write, as shownin FIG. 5. The client 20 issues (1) a write request 54 containing data42 to be written to the volume 18 associated with the remote node 14.The write request 54 is sent to the point of contact node 12 using TCP.

In response to receiving the write request 54, the point of contact node12 preallocates (2) a buffer 36. The point of contact node 12 separatesthe content of the data 42 from the metadata of the data 42, and loadsthe content into the buffer 36. The point of contact node 12 then sends(3) a request 56 that includes the address of the buffer 36 to theremote node 14 via TCP. The request 56 also includes the metadata fromthe data 42. Because the address of the buffer 36 and the metadata arerelatively small, this step can typically be achieved by sending asingle TCP packet, or relatively few TCP packets.

The remote node 14 receives the request 56 and retrieves the location ofthe buffer 36. The remote node then issues (4) an RDMA read command toread the content from the buffer 36. Once the content is received at theremote node 14, the remote node 14 reconstructs the data 42 by combiningthe metadata received in the request 56 with the content retrieved instep (4). The remote node 14 then performs (5) a local write operation58 to write the data 42 to the volume 18 associated with the remote node14.

Once the remote node 14 has finished writing the data, the remote node14 sends (6) an acknowledgement 60 of success to the point of contactnode 12 using TCP. Because the acknowledgement 60 is relatively small,this can typically be achieved using a single packet.

Upon receiving the acknowledgement 60, the point of contact nodedeallocates (7) the buffer 36, and sends (8) an acknowledgement 60 tothe client 20 to inform the client 20 that the write operation has beencarried out successfully.

A particular implementation of these exemplary embodiments is nextdescribed, using TCP for control message and metadata transmissions, andthe SpinNP protocol of the NetApp, Inc. for RDMA communications.Although particular data structures and protocols are described inconnection with FIGS. 6A-10B, these examples are provided forillustration only. The present invention is not limited to the specificexamples provided herein.

Example Implementation

FIGS. 6A-6D depict exemplary data structures suitable for use inexemplary embodiments.

FIG. 6A depicts an exemplary I/O Read Request 62, which may be used (forexample) as the request described in step (3) of FIG. 4. The I/O ReadRequest 62 includes an RDMA header 64 (although the I/O Read Request 62is transmitted using TCP). Within the RDMA header 64, the I/O ReadRequest 62 provides a field 66 for identifying the location of thebuffer 36 in the memory 34 of the point of contact node 12. The RDMAheader 64 also includes a field 68 for providing an identifier of therequested data, such as a file name, ID number, physical address, orsome other identifier that informs the remote node which data is beingrequested.

FIG. 6B depicts an exemplary I/O Read Response 70, which may be used(for example) as the read response described in connection with step (6)of FIG. 4. The I/O Read Response 70 includes a Read Response Header 72which the metadata for the retrieved data 42 is stored. The I/O ReadResponse 70 is transmitted using TCP.

FIG. 6C depicts an exemplary I/O Write Request 74, which may be used(for example) as the Request described in step (3) of FIG. 5. The I/OWrite Request 74 includes an RDMA header 76 (although the I/O WriteRequest 74 is transmitted using TCP). Within the RDMA header 76, the I/OWrite Request 74 provides a field 78 for identifying the location of thebuffer 36 in the memory 34 of the point of contact node 12. The RDMAheader 76 also includes a field 80 for providing the metadata associatedwith the data 42 that is being written to the remote node 14.

FIG. 6D depicts an exemplary I/O Write Response 82, which may be used(for example) as the acknowledgement 60 described in step (6) of FIG. 5.The I/O Write Response 82 includes a read response header 84 thatincludes a flag to indicate whether the write operation was successfullycarried out.

The exemplary data structures of FIGS. 6A-6D may be sent over TCPcommunication paths in the cluster, while data content may betransmitted over RDMA communication paths. Examples of suchcommunication paths are depicted in FIG. 7.

The point of contact node 12 typically communicates with a client usingthe TCP protocol. TCP messages are received by the point of contact node12 and pass through several layers of the network stack. These layersinclude, for example, a drier layer, an LTM layer, a TCP/IP layer, asockets layer, and a PCP layer. The messages are retrieved by a networkmodule (such as server hardware/software) and processed by an NGprotocol 88.

The NG protocol 88 communicates with a cluster session manager (CSM) 88.The CSM 88 of the point of contact node 12 is in communication with apartner CSM 90 on the remote node 14. The CSM 88 and the CSM 90 cancommunicate with each other using TCP by exchanging messages thattraverse the network stack, as illustrated in FIG. 7.

These elements make up the TCP path in the network. Messages describedherein as being sent via TCP may be sent along this path.

The CSM 88 may also communicate with its counterpart CSM 90 using RDMAby sending RDMA messages through an interconnect 96. The interconnect 96establishes an RDMA communications path to a counterpart RDMAinterconnect 98 at the remote node 14. The counterpart RDMA interconnectcommunicates with the CSM 90 of the remote node 14.

These elements make up the RDMA path in the network. Messages describedherein as being sent via RDMA may be sent along this path. Notably, theRDMA path avoids the need to traverse much of the network stack, thusreducing the overhead associated with RDMA transmissions.

At the remote node 14, the CSM may cause read and write operations to beperformed by forwarding requests to SpinHi logic 92, which in turncommunicates with Write Anywhere File Layout (WAFL) logic 94 to readdata from, and write data to, the storage volume 18.

Note that communication between the CSM 90, the SpinHi logic 92, and theWAFL logic 94 do not necessarily rely on any particular networkcommunications protocol such as TCP or RDMA. They may communicate witheach other using any suitable protocol.

The client 20, the point of contact node 12, and the remote node 14 maybe in communication with each other using these communication paths.FIGS. 8-10B describe exemplary methods performed by these entities inorder to effect read and write operations according to exemplaryembodiments.

FIG. 8 depicts an exemplary method performed by a client for requestingthat the cluster perform read and write operations. The method begins atstep 100, in which the client 20 generates an I/O request that instructsthe cluster 10 to perform a read operation or a write operation, asdetermined by the client at step 102.

If the operation is a read operation, processing proceeds to step 104and the client sends an instruction to the point of contact nodespecifying which data the client is requesting for reading. The requestis transmitted using the TCP protocol. Once the request is serviced bythe cluster, the point of contact node 12 transmits the requested datato the client 20, and at step 106 the client receives the requested datavia TCP. Processing then terminates at step 108.

If the determination at step 102 is that the I/O Request is a writeoperation, then processing proceeds to step 110. The client 20constructs a TCP request that encapsulates the data to be written, andtransmits the request to the point of contact node 12. After clientreceives an acknowledgement from the point of contact node 12 that thedata has been successfully written to a volume in the cluster,processing proceeds to step 108 and ends.

FIG. 9A depicts an exemplary method performed by a point of contact node12 in the cluster 10. Processing begins at step 110, in which the pointof contact node 12 receives a client I/O Request via TCP. At step 112,the point of contact node 12 determines whether the request can beserviced locally (i.e., whether the requested data can be read from orwritten to a local storage volume 16 associated with the point ofcontact node 12). If so, then processing proceeds to step 114 and thepoint of contact node performs a local read or write (e.g., using amethod such as the one depicted in FIG. 3A). Processing then terminatesat step 116.

If the determination at step 112 is “no” (i.e., the I/O request cannotbe handled locally), then processing proceeds to step 118 and the pointof contact node 12 determines whether the request is a read request or awrite request. The type of request may be specified in the requestcommunication received at step 110. If the request is a read request,then processing proceeds to step 120. If the request is a write request,then processing proceeds to step 122.

Step 120 is described in more detail with respect to FIG. 9B. Processingfirst proceeds to step 124, where the point of contact node 12 comparesthe size of the data to be read to a predetermined threshold amount. Forexample, the size of the data to be read may be specified in the requestreceived by the point of contact node 12 at step 110, or the point ofcontact node 12 may maintain a directory indicating the size of filesstored on the cluster. The determination made at step 124 is used toevaluate whether to perform the request using RDMA or TCP. The amount ofnetwork overhead that is saved by using RDMA as compared to TCP isdependent on how many times the point of contact node 12 can avoidtraversing the full network stack (which is itself dependent on numberof packets to be transmitted as defined by the size of the requesteddata compared to the network's MTU size). The amount of overheadincurred (and memory resources used) by allocating buffers for an RDMAtransfer must also be considered. Accordingly, the predeterminedthreshold is set at a level that balances these considerations, toensure that relatively small requests continue to be handled via TCP,whereas relatively large requests are handled by RDMA. The presentinventors have achieved throughput savings of about 30% by employing anRDMA transfer for data as small as 32 kB. Throughput is improved, thoughto a smaller degree, with a threshold set at 16 kB. Accordingly, in anexemplary embodiment, the predetermined threshold is set to about 16kB.’

If the determination at step 124 is “no” (i.e., the request size is notlarger than the predetermined threshold), then processing proceeds tostep 126 and the request is handled via TCP. At step 126, the readrequest is forwarded to the remote node on which the data resides, andthe data is received via TCP at step 128. At step 130, the data isreturned to the client via TCP.

If, on the other hand, the determination at step 124 is “yes” (i.e., therequest size is larger than the predetermined threshold), thenprocessing proceeds to step 132 and the point of contact node 12evaluates whether it is possible to communicate with the remote node 14using RDMA. It may be the case, for example, that the remote node doesnot support RDMA, or that the remote node does support RDMA but the RDMAcommunication path (see FIG. 7) is currently constrained. If thedetermination at step 132 is “no,” then processing proceeds to step 126and the read request is handled using TCP.

If the determination at step 132 is “yes” (i.e., the use of RDMA ispossible), then processing proceeds to step 134. At step 134, the pointof contact node preallocates a buffer 36 at a location in its memory 34.The buffer 36 may be preallocated at a size depending on the size of thedata to be read, as identified in step 124.

At step 136, the point of contact node 12 generates an I/O Read Request62 and stores the location of the buffer 36 in the PoC Node BufferLocation field 66. The point of contact node 12 also loads an identifierfor the requested data (e.g., an identifier as specified in the originalrequest from the client 20, or an internal identifier used by thecluster 10) into the requested data ID field 68. The point of contactnode 12 then transmits the I/O Read Request 62 to the remote node usingTCP at step 138 and awaits a response.

At step 140, the point of contact node 12 receives an I/O Read Response70 from the remote node 14 via TCP. At step 142, the point of contactnode 12 retrieves the content of the requested data from the bufferallocated at step 134. At step 144, the point of contact node 12retrieves the metadata associated with the data from the Read ResponseHeader 72 of the I/O Read Response 70, and combines the content with themetadata. Processing then proceeds to step 146, where the point ofcontact node 12 transmits the requested data to the client 20 via TCP.

Optionally, if an error occurs at step 140 such that the read operationfailed, the point of contact node 12 may retry the operation using TCP(step 126).

This concludes the requested read operation.

If the requested operation was a write operation, then processingproceeds from step 118 to 122. Step 122 is described in more detail withrespect to FIG. 9C.

Processing begins at step 148, in which the size of the data to bewritten is compared to a predetermined threshold. The predeterminedthreshold may be the threshold as previously described in connectionwith step 124 of FIG. 9B. If the requested size is not above thepredetermined threshold, then processing proceeds to step 150 and thewrite request is handled using TCP. Otherwise, processing proceeds tostep 152, where it is determined whether an RDMA transfer is possible.Step 152 may be carried out in the same manner as step 132 from FIG. 9B.If not, processing proceeds to step 150 and the write request is handledusing TCP.

If the determination at steps 148 and 152 are both “yes,” thenprocessing proceeds to step 154. The point of contact node 12preallocates a buffer 36 in the memory 34 of the point of contact node12 for holding the data to be written. At step 156, the point of contactnode 12 separates the data into content and metadata, and loads thecontent of the data into the buffer.

At step 158, the point of contact node 12 generates an I/O Write Request74 and loads the location of the buffer 36 into PoC Node Buffer Locationfield 78 of the RDMA Header 76. The metadata from the data is loadedinto the metadata field 80. At step 160, the point of contact node 12transmits the I/O Write Request to the remote node 14 via TCP and awaitsa response.

At step 162, the point of contact node receives an I/O Write Response 82via TCP, and checks the success flag 84 to ensure that the data has beensuccessfully written. If the data has not been successfully written, thepoint of contact node may retry the write operation using TCP (step150).

If the data has been successfully written, then processing proceeds tostep 164, where the point of contact node deallocates the buffer 36. Thepoint of contact node 12 then transmits an acknowledgement of asuccessful write operation to the client 20 at step 168.

FIGS. 10A and 10B describe operations performed by the remote node 14 inresponse to the operations described in FIGS. 9B, and 9C, respectively.FIG. 10A depicts a remote node read method, while FIG. 10B depicts aremote node write method.

In the read operation performed by the remote node 14, processing beginsat step 170 when the remote node 14 receives an I/O Read Request 62 viaTCP. The remote node reads the PoC Buffer Location field 66 to retrievethe location of the buffer 36 on the point of contact node 12 which isintended to store the data.

At step 172, the remote node 14 retrieves the Requested Data ID 68 fromthe I/O Read Request 62 and reads the corresponding data 42 from thevolume 18 associated with the remote node 14. The remote node 14 dividesthe data 42 into content and metadata. At step 174, the remote node 14issues an RDMA write command to write the retrieved content into thepoint of contact node 12 buffer 36 at the location identified in step170.

Once the data has been transferred via RDMA, the remote node 14generates an I/O Read Response 70 by loading the metadata into the ReadResponse Header 72 at step 176. At step 178, the remote node transmitsthe I/O Read Response 70 to the point of contact node 12 via TCP.Processing then terminates at step 180.

In the write operation performed by the remote node 14, processingbegins at step 182 when the remote node 14 receives an I/O Write Request74 via TCP. The remote node 14 reads the PoC Buffer Location field 78 toretrieve the location of the buffer 36 on the point of contact node 12which is storing the content for the data that is intended to bewritten. The remote node also retrieves the metadata for the data fromthe metadata field 80 of the I/O Write Request 74.

At step 184, the remote node 14 issues an RDMA read command to read thecontent of the data from the buffer 36 of the point of contact node 12.At step 186, the content is combined with the metadata retrieved at step182 to recreate the data to be written.

At step 188, the remote node 14 performs a local write operation towrite the combined data to the storage volume 18 associated with theremote node 14. Depending on whether the write is successful, the remotenode 14 generates an I/O Write Response Message 82 with the Success Flagin the Write Response Header 84 set to the appropriate value. At step192, the remote node 14 transmits the I/O Write Response 82 to the pointof contact node 12 via TCP. Processing then proceeds to step 194 andterminates.

Although exemplary embodiments have been described with reference tospecific examples, one of ordinary skill in the art will recognize thatthe present invention is not so limited. Alternative embodiments mayemploy more, fewer, or different components than the apparatuses andsystems described herein. Alternative embodiments may also employ more,fewer, or different steps than the methods described herein, and maycarry out steps in a different order than described. Moreover, althoughexemplary embodiments involve direct communication between the point ofcontact node 12 and the remote node 14, it is understood that thecommunication may occur through one or more intervening nodes, which mayalso employ RDMA to improve network throughput.

1. A method comprising: receiving, via circuitry, a request from a pointof contact node to write data to a remote volume associated with aremote node, wherein the remote node is capable of communicating withthe point of contact node using a first protocol that traverses anetwork stack and a second protocol that bypasses at least a portion ofthe network stack; receiving at least a portion of the data to bewritten via the second protocol that bypasses at least a portion of thenetwork stack; writing the received data to the remote volume; andtransmitting a response to the point of contact node via the firstprotocol that traverses the network stack.
 2. The method of claim 1,wherein the first protocol is transport control protocol (TCP) and thesecond protocol is the remote direct memory access (RDMA) protocol. 3.The method of claim 1, further comprising retrieving, from the request,a location of a preallocated buffer on the point of contact node,wherein receiving the portion of the requested data comprises readingthe portion from the preallocated buffer.
 4. The method of claim 1,wherein: the data comprises metadata and content, receiving the requestfrom the point of contact node comprises receiving the metadata via thefirst protocol, receiving the portion of the requested data comprisesreceiving the content via the second protocol, and writing the receiveddata comprises combining the content with the metadata to reassemble thedata.
 5. The method of claim 1, further comprising receiving the portionof the requested data via the second protocol when a size of therequested data exceeds a predetermined threshold.
 6. The method of claim5, wherein the predetermined threshold is between about 16 kilobytes andabout 32 kilobytes.
 7. The method of claim 1, further comprising:reverting to data transmission via the first protocol when communicationvia the second protocol is not possible.
 8. The method of claim 1,wherein receiving the request from the point of contact node comprisesreceiving the request via the first protocol.
 9. A non-transitorycomputer readable medium storing instructions that, when executed, causecircuitry of a computing device to: receive a request from a point ofcontact node to write data to a remote volume associated with a remotenode; receive at least a portion of the data to be written via abypassing protocol that bypasses at least a portion of a network stack;write the received data to the remote volume; and transmit a response tothe point of contact node via a traversing protocol that traverses thenetwork stack.
 10. The medium of claim 9, wherein the traversingprotocol is transport control protocol (TCP) and the bypassing protocolis the remote direct memory access (RDMA) protocol.
 11. The medium ofclaim 9, further storing instructions to retrieve, from the request, alocation of a preallocated buffer on the point of contact node, whereinreceiving the portion of the requested data comprises reading theportion from the preallocated buffer.
 12. The medium of claim 9,wherein: the data comprises metadata and content, receiving the requestfrom the point of contact node comprises receiving the metadata via thetraversing protocol receiving the portion of the requested datacomprises receiving the content via the bypassing protocol, and writingthe received data comprises combining the content with the metadata toreassemble the requested data.
 13. The medium of claim 9, furtherstoring instructions to receive the portion of the data via thebypassing protocol when a size of the data exceeds a predeterminedthreshold.
 14. The medium of claim 9, further storing instructions to:revert to data transmission via the traversing protocol whencommunication via the bypassing protocol is not possible.
 15. The mediumof claim 9, wherein receiving the request at the remote node comprisesreceiving the request via the traversing protocol.
 16. An apparatus,comprising: a network interface capable of communicating with a point ofcontact node using a first protocol that traverses a network stack and asecond protocol that bypasses at least a portion of the network stack;and logic, at least a portion of which is implemented in hardware, thelogic comprising: a request handler configured to receive a request fromthe point of contact node to write data to a remote volume associatedwith the remote node, reception logic configured to receive a portion ofthe data via the second protocol, writing logic configured to write thereceived data to the remote volume, and transmission logic configured totransmit a response to the point of contact node via a traversingprotocol that traverses the network stack.
 17. The apparatus of claim16, wherein the first protocol is transport control protocol (TCP) andthe second protocol is the remote direct memory access (RDMA) protocol.18. The apparatus of claim 16, the logic configured to read the portionof the data from a preallocated buffer in a memory of the point ofcontact node, wherein a location of the preallocated buffer is receivedwith the request.
 19. The apparatus of claim 16, wherein the datacomprises metadata and content, and the reception logic is configured toreceive the content via the second protocol, receive the metadata aspart of the request, and combine the content with the metadata toreassemble the requested data.
 20. The system of claim 16, the logicconfigured to receive the portion of the requested data via the secondprotocol when the requested data exceeds a predetermined threshold.